Introduction to Phoronix Test Suite

Introduction

According to the Phoronix Test Suite website, PTS

is the most comprehensive testing and benchmarking platform available that provides an extensible framework for which new tests can be easily added. The software is designed to effectively carry out both qualitative and quantitative benchmarks in a clean, reproducible, and easy-to-use manner. The Phoronix Test Suite can be used for simply comparing your computer's performance with your friends and colleagues or can be used within your organization for internal quality assurance purposes, hardware validation, and continuous integration / performance management.

Among the strengths of Phoronix Test Suite are its ease of use, the large selection of tests available, the ability to automatically process results and, present them graphically. The software also integrates the OpenBenchmarking.org infrastructure, where users can review information on available tests and test suites, upload results to share these results with other users, and access previously uploaded results and the results of other users for comparison of results.

Also available is Phoromatic, a tool "that allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems all through an intuitive, easy-to-use web interface." I do not cover it here as it is beyonf the scope of this article.

Installation and Configuration Overview

Download

Phoronix Test Suite as a native package in distributions' repositories to be installed using their system package managers. Arch and derivatives have the latest stable version and the latest development version in the AUR; openSUSE Tumbleweed has the latest stable version; others distributions have outdated versions as follows from least outdated to most outdated: Fedora, Ubuntu, Debian.

For those not on Arch based distributions or openSUSE Tumbleweed, PTS is available from the PTS download page as a universal .tar.gz tarball for Unix-like systems which includes an installer script. A .deb package is also available.

The Phoronix Test Suite Download Page

PTS is available from the PTS download page as a universal .tar.gz tarball for Unix-like systems which includes an installer script. But there is no script for uninstalling.

Installation

System-wide installation is not necessary; the .tar.gz compressed archive can be extracted anywhere and the software can be run from the extracted directory in a terminal. I did this for my initial experimentation with PTS from a data directory on the second -- mechanical -- hard drive on the Acer VN7-592G-70EN laptop. This only gave access to the data partition and other native Linux partitions on the secondary hard drive. It could not access the NTFS partition on the secondary hard drive or the root file system on the primary SSD.

Running PTS Locally from the Extracted Tarball

System-wide installation is not necessary, the .tar.gz can be extracted anywhere and the software can be run from the extracted directory in a terminal.

Running PTS in this mode created a hidden file in my user directory where configuration files, test results, and other resources are stored.

I suppose I could have extracted the .tar.gz in the root partition and run the program with root privileges. Instead, I later chose to delete the user directory and start again with the provided install-sh script to install PTS system-wide.

Installation to the system from the downloaded .tar.gz can be performed by running the included script -- install-sh -- with elevated privileges.

When PTS is run for the first time, whether locally or from a system-wide installation, a hidden directory -- /home/username/.phoronix-test-suite is created, containing the configuration of PTS, including installed tests and a directory for saving results. When run with elevated privileges using sudo (in order for tests to be able to write to / PTS creates and accesses /var/lib/phoronix-test-suite/.

Configuration

Configuration of PTS for per-user execution is performed by editing the main configuration file at ~/.phoronix-test-suite/user-config.xml. Among the configuration options that are available through this file are some OpenBenchmarking.org integration items, the default browser used to view results, whether system logs should be saved, and whether test logs should be saved.

Other options useful to those who might use PTS to publish benchmarking results are at ~/.phoronix-test-suite/graph-config.json.

Phoronix Test Suite Configuration

The browser window shows a page of the the full Phoronix Test Suite Documentation as a PDF document listing file locations. The first two items in the list are the files users can use to configure the basic operation of PTS.

Unfortunately the Phoronix Test Suite documentation does not mention that configuration of PTS, when using it with elevated privileges doesn't access files in ~/.phoronix-test-suite but the analogue for the root user /var/lib/phoronix-test-suite. Complicating things is that the configuration for privileged execution of PTS is not in the analogous resource directory, but in the typical location for system-wide configuration, /etc at /etc/phoronix-test-suite.xml. (One might expect a similarly named configuration file to /var/lib/phoronix-test-suite/phoronix-test-suite.xml). The same configuration items mentioned above for per-user configuration in ~/.phoronix-test-suite/user-config.xml can be specified in /etc/phoronix-test-suite.xml for execution with elevated permissions.

As mentioned above, resources for running PTS as root are located in /var/lib/phoronix-test-suite/phoronix-test-suite.xml, such as subdirectories for installed tests and subdirectory for storing test results. To change some of these directories, the configuration file /etc/phoronix-test-suite.xml can be modified. The options I changed were those that would allow me to easily share downloaded tests and saved test results among the different GNU/Linux installations on this computer. I modified the location of the download cache -- which can be shared among different PTS clients in various ways, including on external USB media -- and the location of saved test results, to a directory on another partition accessible to all GNU/Linux installations on this computer.

In addition to the configuration files above a multitude of environment variables are available for more advanced configuration of PTS. For the purposes of this introductory article, I did not pursue these options.

Initial Tests

One of the first tests I ran was executed with

phoronix-test-suite benchmark stress-ng

The above illustrates the command structure of PTS and the operation for performing basic tests. Commands are in the form of

phoronix-test-suite command [object-of-command] [object-of-command] ...

In this case stress-ng is a test profile, or simply a "test". If a particular test profile is not installed -- none were installed by default -- when using the benchmark command, PTS downloads it from the OpenBenchmarking.org repositories and installs it before using it to perform the test(s).

Tests can have numerous options that the user is prompted to specify, in basic executions of PTS. In more advanced use, options can be specified before hand in configuration such that the user is not prompeted. If all options are selected, tests could take many hours.

To get aclimated to PTS, I ran the Flexible IO Tester profile in order to determine the impact of Plasma's Baloo using:

phoronix-test-suite benchmark fio

then when prompted, choosing options to minimize the time to completion of the test. I ran this test twice, once with Baloo running, and once after killing Baloo's processes and saving the results to the same name so that the two tests would be automatically merged. It should be noted that the results of these and other tests mentioned in this article are spurious, not because of anything to do with Phoronix Test Suite, but because I performed the tests while using the computer during the tests, as these testing activities were only to get accustomed to using PTS.

PTS Provides a System Overview with the Test Results

The hardware components, kernel version, compiler options, IO scheduler, and other relevant information regarding the system are provided.

PTS Provides a Tabular Summary of the Test Results

In the presentation of multiple tests saved to the same result name the "winner" is highlighted. The spurious nature of the results is evident from this summary.

Documentation

The documentation provided by Phoronix Test Suite is essential in order to use the software effectively. I'm sure my cursory perusal is not enough to get the most out of PTS. The PDF version of the user guide is available at http://phoronix-test-suite.com/documentation/phoronix-test-suite.pdf. An HTML version is also included in the tarball and copied to /usr/share/doc/phoronix-test-suite/. Some older documents that cover more advanced topics, such as writing custom modules, are also copied to this location.

Problems

The most problematic part of my experience with Phoronix Test Suite has been the OpenBenchmarking.org component. Although not necessary to use PTS, OpenBenchmarking.org is useful for storing test result data, sharing test profiles and test suites, and sharing results. The problems involved registering for an account at OpenBenchmarking.org and logging in to the account. First, although the site supports encrypted connections, it is not required. So if you type "openbenchmarking.org" into a browser, as I did at first, it provides an insecure connection. If you then click "Register," it again connects insecurely instead of converting the connection for the registration. Most sites do this automatically. However, I later learned that if you specifically request the secure site, by including the "https://" clicking the "register" link will open a secure registration page.

Second, in order to successfully log in top the site, the "Auto Log-In / Remember Me" check-box under the username and password entry fields must be activated. Otherwise, it simply opens the main page without logging the user into the account.

More important, although not necessary, as Phoronix Media provides enough, was that I missed resources for inexperienced users to better select test profiles that they might want to use. The only way for a user to select a test is by the name of a test then reading the brief introduction to the test on its page. For example the stress-ng page describes the test as "Stress-NG is a Linux stress tool developed by Colin King of Canonical."

The OpenBenchmarking.org Workstation Test Suite Page

This page shows the basic command to run this test and the tests included in this test suite, but not much else. But clicking the links for the individual tests does open a page that shows the scripts and configuration files that make the test.

There is a command however,

phoronix-test-suite list-recommended-tests

that will list recommended tests organized by category. Still, there are many choices on this list, and it may be difficult to choose the most appropriate test considering the tests themselves may take a long time to complete.

To actually know which tests to use may require looking at the page for each of the listed tests and clicking the link of each individual test, if it is part of a suite, or clicking the link of the latest version of the test, if it is not part of a suite and going through the displayed components that actually comprise the test.

The OpenBenchmarking.org specviewperf Test Profile Page

This page shows the files that control the download and installation of the test profile, as well as the files that define the test and the results collection process.

When clicking one of these links, the opened page will provide listings of the files that control the download and installation of the test profile, as well as the files that define the test and the results collection process. The motivated user can then research the test using the information contained in these files.

One such piece of information is a link to the test profile developer's site. For example, the specviewperf test profile's downloads.xml file indicates a domain of spec.org and the stress-ng test profile's test-definition.xml file indicates the project's URL as http://kernel.ubuntu.com/~cking/stress-ng/. Following this link provides very detailed information on the test and some caveats.

Conclusion

Phoronix Test Suite to be extremely beneficial to GNU/Linux distribution review publishing sites, such as this one, to any GNU/Linux enthusiast who wants to understand system performance, and of course to its intended target, enterprise customers. Making this software freely available to anyone interested in benchmarking and better understanding their systems is a great service. I also believe PTS provides what Phoronix Media promises in its description of the software.